Information processing device, method of information processing, and image display system

ABSTRACT

[Object] To process an image captured by a camera mounted on a moving object device or a user to be suitable for display on an image display device fixed to the head or face of the user. 
     [Solution] The image rendering processor  1042,  in the beginning, corrects the user&#39;s head posture angle q h  using the camera posture angle q c  (F 1421 ), and clips a display angle of view depending on the corrected user&#39;s head posture angle q h * from the captured image to render a free viewpoint image (F 1422 ). Then, the image processing device  1040  transmits the free viewpoint image rendered by the image rendering processor  1042  to the display device  1020  via the communication unit  1041,  and the image is displayed on the display device  1020  (F 1430 ).

TECHNICAL FIELD

The technology disclosed herein relates to an information processingdevice, method of information processing, and image display system,allowing an image captured by a camera that is mounted on a movingobject device or a user to be processed.

BACKGROUND ART

An image display device, i.e., head-mounted display, fixed to the heador face of a user who observes an image is known. The head-mounteddisplay is provided with an image display unit on both the right andleft eyes and is configured to be capable of controlling visual andauditory perception using together with a headphone. The configurationfor blocking the outside world entirely when it is worn on the headaugments virtual reality during viewing. Furthermore, the head-mounteddisplay is capable of projecting a different video image onto each ofthe left and right eyes and is capable of presenting a 3D image bydisplaying an image having parallax on the left and right eyes.

This type of head-mounted display forms a virtual image on the retina ofthe eye to allow the user to observe it. In this regard, the virtualimage is formed on the side of an object in a case where the object issituated closer to the lens than the focal length. In one example, thereis developed a head-mounted display that forms an enlarged virtual imageof a displayed image on the user's pupil by placing a virtual imageoptical system of a wide viewing angle to be spaced by 25 millimetersaway from the front of the pupil and by placing a display panel having asize of the effective pixel range of about 0.7 inches further in frontof the wide viewing angle optical system (e.g. see Patent Literature 1).

Moreover, the user is able to observe an image obtained by clipping apart of the wide-angle image using this type of head-mounted display. Inone example, there is developed a head-mounted display that allowsreality experience of a video image of the whole space of 360 degrees tobe achieved by installing a head motion tracking device including a gyrosensor or the like on the head and by causing it to track the movementof the user's head (see Patent Literatures 2 and 3). The movement of adisplay region in the wide-angle image to cancel the head's movementdetected by a gyro sensor makes it possible to reproduce an image thattracks the head's movement, and thus the user experiences as if he looksa panoramic view of the entire space.

Furthermore, there are also known first person view (FPV) techniquesthat pilot while viewing a first person viewpoint (pilot viewpoint)image captured by a wireless camera mounted on a radio-controlled devicesuch as a helicopter. In one example, a moving object control devicecomposed of a moving object equipped with an image capturing device anda wearable PC for remotely controlling the moving object through anoperator's operation is developed (e.g. see Patent Literature 4). Themoving object side receives a signal for controlling the moving object'soperation to control its own operation, receives a signal forcontrolling an image capturing device equipped thereon to control theimage capturing operation, and transmits video and audio signalsoutputted from the image capturing device to the wearable PC. Meanwhile,the wearable PC side generates a signal for controlling an operation ofthe moving object in response to the operator's operation, generates asignal for controlling an operation of the image capturing device inresponse to the operator's voice. The wearable PC side transmitswirelessly the generated signal to the moving object, receiveswirelessly a signal outputted from the image capturing device,reproduces a video signal, and displays it on a monitor screen.

Furthermore, a network system is developed in which a radio-controlledcar equipped with a three-dimensional stereo camera for medium-to-longdistance and a three-dimensional stereo camera for short distancetransmits a three-dimensional composite image to be displayed on acontroller side (e.g. see Patent Literature 5). A network system isdeveloped in which a model device captures an image of an area in frontof the model device and a controller side receives information on theimage, position, and direction of the model device, generates a virtualimage based on the position and direction, and displays it (e.g. seePatent Literature 6).

CITATION LIST Patent Literature

Patent Literature 1: JP 2012-141461A

Patent Literature 2: JP H9-106322A

Patent Literature 3: JP 2010-256534A

Patent Literature 4: JP 2001-209426A

Patent Literature 5: JP 2012-151800A

Patent Literature 6: JP 2012-143447A

DISCLOSURE OF INVENTION

Technical Problem

An object of the technology disclosed herein is to provide an improvedinformation processing device, method of information processing, andimage display system, capable of processing preferably an image capturedby a camera that is mounted on a moving object device or a user.

Furthermore, another object of the technology disclosed herein is toprovide an improved information processing device, method of informationprocessing, and image display system, capable of processing an imagecaptured by a camera to be displayed suitably on an image display devicethat is fixed to the user's head or face.

Solution to Problem

The technology disclosed herein has been devised in view of the problem,a first aspect thereof is an information processing device including: ahead posture acquisition unit configured to acquire information on headposture of a user; a camera posture acquisition unit configured toacquire information on posture of a camera; and an image renderingprocessor configured to generate an image to be displayed on a displaydevice from an image captured by the camera based on the head posture ofthe user and the posture of the camera, the display device being fixedto a head or a face of the user.

According to a second aspect of the technology disclosed herein, tin theinformation processing device according to the first aspect, the camerais mounted on a moving object device.

According to a third aspect of the technology disclosed herein, thecamera of the information processing device according to any one of thefirst and second aspects is configured to capture an omnidirectionalimage or a wide-angle image, and the image rendering processor isconfigured to correct the head posture of the user using the posture ofthe camera in capturing and generate an image obtained by clipping anangle of view depending on the corrected head posture of the user fromthe image captured by the camera.

According to a fourth aspect of the technology disclosed herein, in theinformation processing device according to the first aspect, the camerais fixedly mounted on the head or the face of the user.

According to a fifth aspect of the technology disclosed herein, theimage rendering processor of the information processing device accordingto the fourth aspect is configured to generate an image obtained byclipping an angle of view depending on a first conversion parameter fromthe image captured by the camera, the first conversion parameter beingused to convert the posture of the camera to the head posture of theuser.

According to a sixth aspect of the technology disclosed herein, theimage rendering processor of the information processing device accordingto the fifth aspect is configured to perform image generation using thehead posture of the user at a time when the camera performs capturingand the head posture of the user predicted after a lapse of delay timeuntil an image is displayed on the display device.

According to a seventh aspect of the technology disclosed herein, theimage rendering processor of the information processing device accordingto the first aspect is configured to construct a three-dimensional modelof surrounding environment based on data of time-series image capturedby the camera, estimate a current position of the camera in thethree-dimensional model, predict a position and posture of an eye of theuser after a lapse of delay time from capturing by the camera todisplaying on the display device using a second conversion parameterused to convert a position and posture of the camera to the position andposture of the eye of the user, and generate an image captured in apredicted position and posture of the eye from the three-dimensionalmodel.

According to an eighth aspect of the technology disclosed herein, theinformation processing device according to the first aspect furtherincludes: a controller configured to remotely operate the moving objectdevice; and a filter configured to constrain a trajectory of the movingobject device and cut off an input other than the constrained trajectoryfrom the controller. The input from the controller is configured to beconverted to a position command, a velocity command, or an accelerationcommand in a direction along the constrained trajectory for transmittingto the moving object device.

According to a ninth aspect of the technology disclosed herein, theinformation processing device according to the eighth aspect isconfigured in a manner that, when the trajectory of the moving objectdevice is constrained to a straight motion, the filter cuts off an inputother than a back-and-front direction from the controller, and transmitsa command to maintain a trajectory of the straight motion to the movingobject device.

According to a tenth aspect of the technology disclosed herein, theinformation processing device according to the ninth aspect isconfigured in a manner that, the trajectory is set to a straight linetoward a current direction of travel through a current point of themoving object or to a straight line connecting the current point of themoving object device and a destination point.

According to an eleventh aspect of the technology disclosed herein, theinformation processing device according to the eighth aspect isconfigured in a manner that, when the trajectory of the moving objectdevice is constrained to a circular motion, the filter cuts off an inputother than a left-and-right direction from the controller, and transmitsa command to maintain a trajectory of the circular motion to the movingobject device and controls posture of the moving object device to bedirected toward a destination point.

According to a twelfth aspect of the technology disclosed herein, theinformation processing device according to the eleventh aspect isconfigured in a manner that the trajectory is set to a circular motionpassing through a current point of the moving object device centered ona destination point or to a circular motion in a horizontal planethrough the current point of the moving object device centered on aperpendicular axis intersecting a destination point.

A thirteenth aspect of the technology disclosed herein is a method ofinformation processing, the method including: a head posture acquisitionstep of acquiring information on head posture of a user; a cameraposture acquisition step of acquiring information on posture of acamera; and an image rendering processing step of generating an image tobe displayed on a display device from an image captured by the camerabased on the head posture of the user and the posture of the camera, thedisplay device being fixed to a head or a face of the user.

A fourteenth aspect of the technology disclosed herein is an imagedisplay system including: a camera; a display device used while beingfixed to a head or a face of a user; and an image processing deviceconfigured to generate an image to be displayed on the display devicefrom an image captured by the camera based on the head posture of theuser and the posture of the camera.

While the expression “system” is herein used to refer to a logicalgrouping of a plurality of devices (and/or functional modules thatimplement specific functions), and it does not matter whether eachdevice and/or functional module is included in a single housing.

Advantageous Effects of Invention

According to the technology disclosed herein, it is possible to providean improved information processing device, method of informationprocessing, and image display system, capable of processing preferablyan image captured by a remote camera installed in a moving object or auser to be displayed suitably on an image display device fixed to theuser's head or face.

Note that the advantageous effects described in this specification aremerely for the sake of example, and the advantageous effects of thepresent invention are not limited thereto. Furthermore, in some casesthe present invention may also exhibit additional advantageous effectsother than the above-mentioned advantageous effects.

The other objects, features, and advantages of the technology disclosedherein will be clarified by a more detailed description based on theexemplary embodiments discussed hereinafter and the attached drawings.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating schematically a configuration of animage display system 100 according to an embodiment of the technologydisclosed herein.

FIG. 2 is a diagram illustrating a user, as viewed from the front, whowears a head-mounted display 110 on the head, which is applied to theimage display system illustrated in FIG. 1.

FIG. 3 is a diagram illustrating a user, as viewed from the above, whowears the head-mounted display 110 illustrated in FIG. 2.

FIG. 4 is a diagram illustrating an exemplary configuration of anomnidirectional camera 400.

FIG. 5 is a diagram illustrating an exemplary configuration of theomnidirectional camera 400.

FIG. 6 is a diagram illustrating an exemplary configuration of theomnidirectional camera 400.

FIG. 7 is a diagram illustrated to describe how a user wearing thehead-mounted display 110 views a free viewpoint image.

FIG. 8 is a diagram illustrating an exemplary functional configurationof the image display system 100.

FIG. 9 is a diagram illustrating an example of a quaternion.

FIG. 10 is a diagram illustrating another exemplary functionalconfiguration of the image display system 100.

FIG. 11 is a diagram illustrating how to clip a display angle of viewfrom an omnidirectional image depending on a user's head posture.

FIG. 12 is a diagram illustrating how to clip a display angle of viewthat does not match a user's head posture due to a change in posture ofthe omnidirectional camera.

FIG. 13 is a diagram illustrating how to clip a display angle of viewobtained by correcting a user's head posture depending on a variation inposture of an omnidirectional camera.

FIG. 14 is a diagram illustrating a processing procedure for clipping afree viewpoint image from an omnidirectional image at a display angle ofview obtained by correcting a user's head posture depending on avariation in posture of an omnidirectional camera.

FIG. 15 is a diagram illustrating an exemplary configuration of thevideo see-through image display system 100.

FIG. 16 is a diagram a processing procedure for displaying an image thatmatches a viewing angle of a user by aligning an image captured by acamera with a reference coordinate system of a display optical system.

FIG. 17 is a diagram illustrating a processing procedure for displayingan image that matches a viewing angle of the user by aligning an imagecaptured by a camera with a reference coordinate system of a displayoptical system in consideration of delay time between image capturingand displaying.

FIG. 18 is a diagram illustrating a processing procedure for displayingan image that matches a viewing angle of the user by combiningthree-dimensional reconstruction technique and head motion trackingprediction in consideration of delay time between image capturing anddisplaying.

FIG. 19 is a diagram illustrating how to operate remotely a movingobject device 120 using a controller 130 by restricting the controldegree of freedom of the moving object device 120 to a straight motion.

FIG. 20 is a diagram illustrating a processing procedure for controllingthe remote operation by restricting the control degree of freedom of themoving object device 120 to a straight motion.

FIG. 21 is a diagram illustrating how to operate remotely the movingobject device 120 using the controller 130 by restricting the controldegree of freedom of the moving object device 120 to a circular motion.

FIG. 22 is a diagram illustrating a processing procedure for controllingthe remote operation by restricting the control degree of freedom of themoving object device 120 to a circular motion.

MODE(S) FOR CARRYING OUT THE INVENTION

An embodiment of the technology disclosed herein will be described inmore detail with reference to the drawings. In the followingdescription, an omnidirectional image or an omnidirectional camera isfundamentally used as an illustrative example. However, even in a casewhere a wide-angle image or a wide-angle camera is used, it can betreated similarly to the case of the omnidirectional image byconsidering an omnidirectional image in which other regions than ashooting angle of view are entirely painted with, for example, a colorof black or a camera outputting this image, and thus the technologydisclosed herein has no loss of generality.

FIG. 1 illustrates schematically a configuration of an image displaysystem 100 according to an embodiment of the technology disclosedherein. The illustrated image display system 100 is configured toinclude an image display device 110 (head-mounted display) used whilebeing mounted on the head or face of a user and moving object devices120-1, 120-2, . . . such as aircraft (or, a helicopter and other flyingobjects), motor vehicle, and watercraft. Some of the moving objectdevices 120 may be a radio-controlled device that is operated remotelyby radio, and the user can pilot the radio-controlled device using acontroller 130. Each of the moving object devices 120-1, 120-2, 120-3, .. . is equipped with an omnidirectional camera and captures a landscapeduring movement. The controller 130 may be, in one example, amulti-functional information terminal such as smartphones and tablets,and launches an application for piloting the moving object device 120.

Wireless connection is established between the head-mounted display 110and the moving object device 120 and between the controller 130 and themoving object device 120, in one example, through wireless network,infrared communication, or the like. An image captured by theomnidirectional camera can be transmitted to other devices including thehead-mounted display 110 by using a wireless communication function ofthe moving object device 120. However, for simplicity of description,the omnidirectional camera is herein assumed to be provided with its ownwireless communication function.

FIG. 2 illustrates a user, as viewed from the front, who wears thehead-mounted display 110 on the head, which is applied to the imagedisplay system illustrated in FIG. 1.

The head-mounted display 110, when being worn on the head or face of auser, covers directly the user's eyes, thereby providing a sense ofimmersion for the user who is viewing an image. Furthermore, a displayedimage is invisible to the outside (other people), and thus protection ofprivacy can be easily achieved in displaying information. It isdifferent from a see-through type, and the user who wears thehead-mounted display 110 is unable to view the real-world landscape. Ifit is equipped with an externally mounted camera (not shown) forcapturing a landscape in the user's line-of-sight direction, thecaptured image is displayed on the head-mounted display 110, and thusthe user is able to view indirectly the real-world landscape (i.e., thelandscape is displayed using video see-through visualization).

The head-mounted display 110 illustrated in FIG. 2 is a structuresimilar to a helmet shape, and is configured to cover directly the leftand right eyes of the user wearing it. The head-mounted display 110 hasa display panel (not shown in FIG. 2) that is arranged in the positionthat faces the left and right eyes on the inner side of a main body ofthe head-mounted display 110. The user observes the display panel. Thedisplay panel is composed, in one example, of a micro-display such asorganic electro-luminescence (EL) device and liquid crystal display or alaser scanning display such as retinal direct projection display.

The head-mounted display 110 has a microphone that is installed nearboth left and right ends of the main body of the head-mounted display110. These microphones are located nearly symmetrically on both sidesand allow only voice localized at the center (user's voice) to berecognized and to be separated from the surrounding noise or otherpeople's voice, thereby preventing malfunction in operating it byinputting the voice.

Furthermore, the head-mounted display 110 has a touch panel that isarranged on the outside of the head-mounted display 110. The touch panelallows the user to perform a touch input with the fingertip or the like.Although a pair of left and right side touch panels are provided in theillustrated example, a single or three or more touch panels may beprovided.

FIG. 3 illustrates a user, as viewed from the above, who wears thehead-mounted display 110 illustrated in FIG. 2. The illustratedhead-mounted display 110 has the display panels for the left and righteyes on either side that faces the user's face. The display panel iscomposed, in one example, of a micro-display such as organic EL deviceand liquid crystal display or a laser scanning display such as retinaldirect projection display. An image to be displayed on the display panelis observed, as an enlarged virtual image obtained by passing through avirtual image optical unit, by each of the left and right eyes of theuser. Furthermore, each user has individual difference in height of eyesand pupil distance, and thus the eyes of the user wearing it isnecessary to be aligned with left and right display systems. In theexample illustrated in FIG. 3, a pupil distance adjustment mechanism isprovided between the display panel for the right eye and the displaypanel for the left eye.

The omnidirectional camera to be mounted on the moving object device 120can be configured with a combination of a plurality of video cameras, inone example. FIGS. 4 to 6 illustrate an exemplary configuration of anomnidirectional camera 400 including six video cameras.

The six video cameras 401, 402, . . . , 406 are fixed at theirrespective predetermined positions and output a captured image to animage processor 410 in synchronization with it. Each of the videocameras 401, 402, . . . , 406 employs a complementarymetal-oxide-semiconductor (CMOS) image sensor as their respective imagesensors, in one example.

The image processor 410 generates one omnidirectional image frame (orwide-angle image frame) by stitching images captured by the videocameras 401, 402, . . . , 406 depending on relationship betweenpositions at which the video cameras are located. Some or all of thegenerated omnidirectional images are transmitted wirelessly to thehead-mounted display 110, and are provided as a free viewpoint image inwhich the viewpoint is shifted depending on the posture (line-of-sightdirection) of the head of the user wearing it.

FIGS. 5 and 6 illustrate schematically an exemplary arrangement of thesix video cameras 401, 402, . . . , 406. FIG. 5 is a view looking downfrom the above, and FIG. 6 is a perspective view looking from the side.As illustrated, the six video cameras 401, 402, . . . , 406 are radiallyarranged in back-to-back relation to each other in the direction oftheir respective main axes.

More preferably, the positions of the viewpoint (position of the camera)of the video cameras 401, 402, . . . , 406 are arranged at predeterminedangular intervals on a horizontal concentric circle centered on avertical reference axis 501 (refer to FIGS. 5 and 6) passing through apredetermined reference point (described later). In the presentembodiment, the six video cameras 401, 402, . . . , 406 are arranged at60 degree intervals. Furthermore, the video cameras 401, 402, . . . ,406 are arranged so that both end portions of an image capturing viewingangle overlap between adjacent ones, thereby capturing the entirecircumference without discontinuity in the horizontal direction.

Moreover, a specific exemplary configuration of the omnidirectionalcamera that is applicable to the image display system 100 according tothe present embodiment will be found, in one example, in thespecification of Patent Application No. 2014-128020 that has beenassigned to the present applicant. The technology disclosed herein isnot limited to the configuration of a particular omnidirectional camera.

FIG. 7 illustrates how a user wearing the head-mounted display 110 viewsa free viewpoint image in the image display system 100 according to thepresent embodiment (i.e. how to display an image that tracks themovement of the user's head).

In the user's viewpoint, the depth direction is defined as z_(w) axis,the horizontal direction is defined as y_(w), the vertical direction isdefined as x_(w), and the origin positions of the user's reference axesx_(w), y_(w), and z_(w) are defined as a user's viewpoint position.Thus, roll θ_(z) corresponds to motion around the z_(w) axis of theuser's head, tilt θ_(y) corresponds to motion around the y_(w) axis ofthe user's head, and pan θ_(z) corresponds to motion around the x_(w)axis of the user's head.

In the beginning, a movement in the direction of each of roll, tilt, andpan of the user's head (θ_(z), θ_(y), θ_(z)) or posture informationcomposed of translation of the head are detected. Then, the center of aregion 702 to be clipped is shifted from an original omnidirectionalimage 701 captured by an omnidirectional camera to track the posture ofthe user's head, and an image of the region 702 clipped by apredetermined angle of view at its center position is rendered. Morespecifically, a display region is shifted to cancel a movement of theuser's head by rotating a region 702-1 depending on the roll componentof the user's head motion, by rotating a region 702-2 depending on thetilt component of the user's head motion, or by rotating a region 702-3depending on the pan component of the user's head motion. This makes itpossible for the head-mounted display 110 to present a free viewpointimage that tracks the user's head movement.

Moreover, an example of a process of rendering a free viewpoint imagedepending on the user's head posture from an omnidirectional imagecaptured by an omnidirectional camera includes a method of performing itin the omnidirectional camera, a method of transmitting anomnidirectional image to the head-mounted display 110 and performing itin the head-mounted display 110, and a method of uploading anomnidirectional image to a cloud computer and performing it on a cloud.

In this regard, as illustrated in FIG. 1, in a case where theomnidirectional camera is mounted on the moving object device 120, it isenvisaged that the moving object device 120 changes its course. With achange in the courses of the moving object device 120, the posture ofthe omnidirectional camera varies. Thus, even when the user wearing thehead-mounted display 110 is not intended to move his head, an image tobe viewed will vary disadvantageously. If the user views an unintendedimage that does not match the user's movement, the user will sufferhealth hazard such as virtual reality (VR) sickness.

Thus, in the present embodiment, the head posture of the user who wearsthe head-mounted display 110 is corrected depending on a variation inposture of the omnidirectional camera to perform clipping of a displayedimage from the omnidirectional image. Such a process makes it possiblefor a free viewpoint image at the same place to remain visible as longas the user does not move, thereby preventing VR sickness.

FIG. 8 illustrates an exemplary functional configuration of the imagedisplay system 100 in which the image clipping process as describedabove can be implemented. The illustrated image display system 100 isconfigured to include three devices, that is, a head motion trackingdevice 810, a display device 820, and image capturing device 830.

The head motion tracking device 810 is used while being mounted on thehead of the user who observes an image displayed on the display device820, and outputs information on the user's head posture to the displaydevice 820 at predetermined transmission intervals. In the illustratedexample, the head motion tracking device 810 is configured to include asensor unit 811, a posture angle calculation unit 812, and acommunication unit 813.

The sensor unit 811 is configured with a combination of a plurality ofsensor devices such as gyro sensor, acceleration sensor, and geomagneticsensor, in one example, and is configured to detect a posture angle ofthe user's head. In this description, it is assumed to be a sensor thatis capable of detecting a total of nine axes of three-axis gyro sensor,three-axis acceleration sensor, and three-axis geomagnetic sensor.

The posture angle calculation unit 812 calculates information on theuser's head posture based on a result obtained by detection of nine axesby the sensor unit 811. In the present embodiment, the posture angle isassumed to be represented as a quaternion. Furthermore, in the followingdescription, a three-dimensional vector indicating a position is definedas p, and a quaternion indicating a posture is defined as q. Thequaternion q is a quaternion that consists of an axis of rotation(vector) and an angle of rotation (scalar), as shown in the followingformula (1) and FIG. 9. The quaternion is suitable for computationbecause of no singularity. In the field of computer graphics, anobject's posture is typically represented using a quaternion.

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 1} \right\rbrack & \; \\\begin{matrix}{q = {s + {ix} + {jy} + {kz}}} \\{= \left( {s,x,y,z} \right)} \\{= \left( {{\cos \left( {\theta/2} \right)},{n\; {\sin \left( {\theta/2} \right)}}} \right)}\end{matrix} & (1)\end{matrix}$

The head motion tracking device 810 and the display device 820 areassumed to be interconnected by wireless communication such as Bluetooth(registered trademark) communication. Alternatively, the connectionbetween the head motion tracking device 810 and the display device 820may be established via high-speed wired interface such as universalserial bus (USB) instead of wireless communication. The information onthe user's head posture obtained by the posture angle calculation unit812 is transmitted to the display device 820 via the communication unit813.

The image capturing device 830 is configured to include anomnidirectional camera 831, a sensor unit 832, a posture anglecalculation unit 833, and a communication unit 834. In the presentembodiment, the image capturing device 830 is used while being mountedon the moving object device 120.

The omnidirectional camera 831 is configured as illustrated in FIGS. 4and 8, and captures an omnidirectional image. A detailed descriptionthereof will be omitted.

The sensor unit 832 is configured with a combination of a plurality ofsensor devices such as gyro sensor, acceleration sensor, and geomagneticsensor, in one example. In this description, it is assumed to be asensor that is capable of detecting a total of nine axes of three-axisgyro sensor, three-axis acceleration sensor, and three-axis geomagneticsensor. The posture angle calculation unit 833 calculates information onposture of the omnidirectional camera 831 based on a result obtained bydetection of nine axes by the sensor unit 832. In the presentembodiment, a posture angle is assumed to be represented as a quaternion(same as the above).

The image capturing device 830 and the display device 820 are assumed tobe interconnected by wireless communication such as wireless fidelity(Wi-Fi). The information on the image captured by the omnidirectionalcamera 831 and the information on the posture of the omnidirectionalcamera 831 obtained by the posture angle calculation unit 833 aretransmitted to the display device 820 via the communication unit 834.

The display device 820 is equivalent to the head-mounted display 110 inthe image display system 100 illustrated in FIG. 1. In the exampleillustrated in FIG. 8, the head motion tracking device 810 is configuredas a separate device from the display device 820 (in one example, thehead motion tracking device 810 is manufactured and sold as an optionalproduct of the head-mounted display 110). Alternatively, the head motiontracking device 810 and the display device 820 can be integrated intoone head-mounted display 110.

The display device 820 is configured to include a first communicationunit 821, a second communication unit 824, an image rendering processor822, and a display unit 823.

In a case where the display device 820 is configured as a head-mounteddisplay, the display unit 823 is provided with left and right screensthat are respectively fixed to the user's left and right eyes, in oneexample, and displays an image for the left eye and an image for theright eye. The screen of the display unit 823 is composed, in oneexample, of a display panel including a micro-display such as organic ELdevice and liquid crystal display or of a laser scanning display such asretinal direct projection display. Furthermore, a virtual image opticalunit (not illustrated), which enlarges an image displayed on the displayunit 823, projects it, and focuses an enlarged virtual image having apredetermined angle of view on the user's pupil, is provided.

The first communication unit 821 receives the information on the user'shead posture from the head motion tracking device 810 via thecommunication unit 813. Furthermore, the second communication unit 824receives the information on the image captured by the omnidirectionalcamera 831 from the image capturing device 830 via the communicationunit 834 and the information on the posture of the omnidirectionalcamera 831 obtained by the posture angle calculation unit 833. Moreover,in the present embodiment, the posture angle calculation unit 812 isequipped in the head motion tracking device 810 and the posture anglecalculation unit 833 is equipped in the image capturing device. However,it is possible to have a configuration in which the communication units813 and 834 transmit wirelessly results detected by the sensor units 811and 832, respectively, without any modification, instead of transmittingtheir respective posture information items, thereby preventing thedevices 810 and 830 from performing the posture angle calculation, andthe display device 820 performs their respective posture anglecalculation processes using sensor-based information received by thefirst communication unit 821 or the second communication unit 824.

The image rendering processor 822 renders an image obtained by clippinga display angle of view corresponding to the information on the user'shead posture from the omnidirectional image. The shift of a displayangle of view obtained by clipping an original image to cancel a postureangle of the user's head makes it possible for the display unit 823 todisplay an image that tracks the head's movement. Thus, it is possiblefor the user to experience viewing a large screen.

Furthermore, in the present embodiment, the image rendering processor822 corrects the head posture of the user (user wearing the head-mounteddisplay 110) who observes an image displayed on the display device 820depending on the variation in posture of the omnidirectional camera 831,thereby performing clipping of an omnidirectional image. Such acorrection process allows a free viewpoint image at the same place toremain visible as long as the user does not move, even when the postureof the camera 831 varies.

FIG. 10 illustrates another exemplary functional configuration of theimage display system 100. The illustrated image display system 100 isconfigured to include four devices, that is, a head motion trackingdevice 1010, a display device 1020, an image capturing device 1030, andan image processing device 1040.

The head motion tracking device 1010 is used while being mounted on thehead of the user who observes an image displayed on the display device1020, and outputs information on the user's head posture to the imageprocessing device 1040 at predetermined transmission intervals. In theillustrated example, the head motion tracking device 1010 is configuredto include a sensor unit 1011, a posture angle calculation unit 1012,and a communication unit 1013.

The sensor unit 1011 is a sensor capable of detecting nine axes (same atthe above) and detects a posture angle of the user's head. The postureangle calculation unit 1012 calculates a quaternion q_(h) indicating aposture angle of the user's head based on a result obtained by detectionof nine axes by the sensor unit 1011. Then, the calculated quaternionq_(h) is transmitted to the image processing device 1040 via thecommunication unit 1013.

The image capturing device 1030 is configured to include anomnidirectional camera 1301, a sensor unit 1302, a posture anglecalculation unit 1033, and a communication unit 1034. The imagecapturing device 1030 is used while being mounted on the moving objectdevice 120 (same as the above).

The omnidirectional camera 1031 is configured as illustrated in FIGS. 4to 6, and captures an omnidirectional image. The sensor unit 1032 is asensor capable of detecting nine axes (same as the above), and detects aposture angle of the omnidirectional camera 1301. The posture anglecalculation unit 1033 calculates a quaternion q_(c) indicating theposture angle of the omnidirectional camera 1031 based on a resultobtained by detection of nine axes by the sensor unit 1032. Then, thecalculated quaternion q_(c) is transmitted to the image processingdevice 1040 via the communication unit 1034.

The image processing device 1040 is composed of a cloud computer, in oneexample. The image processing device 1040 receives the quaternion q_(h)indicating a posture angle of the user's head from the head motiontracking device 1010 via the communication unit 1041, and receivesinformation on an image captured by the omnidirectional camera 1031 andthe quaternion q_(c) indicating the posture angle of the omnidirectionalcamera 1031 from the image capturing device 1030. Then, the imagerendering processor 1042 renders the image obtained by clipping thedisplay angle of view corresponding to the information on the user'shead posture from the omnidirectional image and transmits it to thedisplay device 1020 via the communication unit 1041. Furthermore, aswith the configuration described above, it is possible to have aconfiguration in which the head motion tracking device 1010 or the imagecapturing device 1030 does not perform the posture angle calculation buttransmits the sensor-based information to the image processing device1040 for allowing the posture angle calculation to be performed by theimage processing device 1040.

The display device 1020 displays the image information, which isreceived from the image processing device 1040 via the communicationunit 1021, on the display unit 1023. The display device 1020 shifts thedisplay angle of view in the original image to cancel the posture angleof the user's head, and thus the display unit 1023 can display an imagethat tracks the head's movement. Thus, it is possible for the user toexperience viewing a large screen. The image processing device 1040 iscomposed, in one example, of a cloud computer, and in the case where itis configured to be connected to the display device 1020 via acommunication unit, a transmission delay in the communication unit maybecome a problem. In a modified example of the system configurationillustrated in FIG. 10, the display device 1020 is provided with asecond image rendering processor (not illustrated) and the imageprocessing device 1040 transmits not only information on the clippedimage but also information on the user's head posture used in clippingsimultaneously, and thus the display device 1020 can adjust the positionat which the image is displayed using the posture information receivedfrom the image processing device 1040 and posture information that isnewly received from the head motion tracking device 1010, therebyalleviating the transmission delay problem.

FIG. 11 illustrates how to clip a display angle of view 1101corresponding to the head posture of the user who wears the head-mounteddisplay 110 from an omnidirectional image 1100 captured by anomnidirectional camera mounted on the moving object device 120.

Furthermore, FIG. 12 illustrates how the posture of the moving objectdevice 120, that is, the omnidirectional camera is changed in a leftwarddirection even though the user does not move his head posture, andaccordingly, a display angle of view 1201 obtained by clipping from anomnidirectional image 1200 moves in a rightward direction. In this case,even though the user wearing the head-mounted display 110 does not movehis head, a phenomenon occurs in which an image being viewed varies. Ifan unintended image that does not match the user's movement is viewed,the user will suffer health hazard such as virtual reality (VR)sickness.

In one example, in the case where the user operates remotely the movingobject device 120 using the controller 130 while the user views an imagecaptured by a camera mounted on the moving object device 120, if themoving object device 120 performs a motion against the user's intentionor an unintended motion, the user is likely to experience VR sickness.Even if the moving object device 120 moves according to the remoteoperation by the user, a vigorous movement of a displayed image is morelikely to cause VR sickness.

In order to prevent the VR sickness as described above, there may beconceived a solution that causes a low-pass filter to cut it byimproving the control performance of the moving object device 120 by thecontroller 130 so that motion against the user's intention is preventedfrom occurring, that is, the moving object device 120 performs onlyslow-speed motion, or alternatively so that fast-speed motion of acamera is prevented from being displayed on the head-mounted display110.

Meanwhile, in the technology disclosed herein, an image captured by acamera mounted on the moving object device 120 is displayed by clippinga portion suitable for the direction viewed by the user wearing thehead-mounted display 110 from an omnidirectional image, rather thanbeing displayed without any modification.

In one example, in the embodiment illustrated in FIGS. 8 and 10, theuser's head posture measured by the head motion tracking device 810 or1010 is corrected depending on a variation in the posture of theomnidirectional camera 830 or 1030. FIG. 13 illustrates how to clip adisplay angle of view 1301 obtained by correcting a user's head posturedepending on a variation in posture of an omnidirectional camera from anomnidirectional image 1300 captured by an omnidirectional camera whoseposture is changed in a leftward direction. Such a process makes itpossible for the same free viewpoint image to remain visible as long asthe user does not move even if the posture of the omnidirectional camerais changed, thereby preventing VR sickness.

In clipping a display angle of view suitable for the direction beingviewed by the user from an omnidirectional image (or a wide-angleimage), rather than using a coordinate system fixed to a camera and acoordinate system fixed to the user's body, a third coordinate system isused. In other words, the camera side and the head-mounted display sideperform independently estimation of the position and posture variation,and a region of the image to be displayed is determined based on aresult obtained from both sides.

FIG. 14 illustrates a processing procedure for clipping a free viewpointimage from an omnidirectional image at a display angle of view obtainedby correcting a head posture of the user wearing the head-mounteddisplay 110 depending on a variation in posture of an omnidirectionalcamera. In one example, this processing procedure is performed by theimage rendering processor 822 included in the display device 820 in theimage display system 100 illustrated in FIG. 8, or is performed by theimage processing device 1040 in the image display system 100 illustratedin FIG. 10. In the following, for convenience' sake, the descriptionwill be given on the assumption that the process is performed in theimage display system 100 illustrated in FIG. 10.

The omnidirectional camera 1031 of the image capturing device 1030mounted on the moving object device 120 captures an omnidirectionalimage (F1401). Furthermore, the sensor unit 1032 detects a posture angleof the omnidirectional camera 1031, and the posture angle calculationunit 1033 calculates a quaternion q_(c) indicating a posture angle ofthe omnidirectional camera 1031 based on a result obtained by detectionof nine axes by the sensor unit 1032 (F1402).

Then, the captured image and the camera posture angle q_(c) aretransmitted to the image processing device 1040 via the communicationunit 1034.

On the other hand, in the head motion tracking device 1010, the sensorunit 1011 detects a posture angle of the user's head, and the postureangle calculation unit 1012 calculates a quaternion q_(h) indicating aposture angle of the user's head based on a result obtained by detectionin the sensor unit 1011 (F1411). Then, the head posture angle q_(h) istransmitted to the image processing device 1040 via the communicationunit 1013.

In the image processing device 1040, the communication unit 1041receives the captured image and the camera posture angle q_(c) from theimage capturing device 1030 and receives the user's head posture angleq_(h) from the head motion tracking device 1010. In the case where thecamera posture angle q_(c) is not changed, the image rendering processor1042 may clip a display angle of view depending on the user's headposture angle q_(h) from the captured image to render the free viewpointimage. However, in the present embodiment, it is envisaged that thecamera posture angle q_(c) varies as illustrated in FIG. 13. Thus, inthe beginning, the image rendering processor 1042 corrects the user'shead posture angle q_(h) using the camera posture angle q_(c) (F1421),and clips a display angle of view depending on the corrected user's headposture angle q_(h)* from the captured image to render a free viewpointimage (F1422). Then, the image processing device 1040 transmits the freeviewpoint image rendered by the image rendering processor 1042 to thedisplay device 1020 via the communication unit 1041, and the displaydevice 1020 displays it (F1430).

Moreover, the process of correcting the corrected user's head postureangle q_(h) in the above process F1422 is performed in accordance withthe following formula (2). In other words, the corrected user's headposture angle q_(h)* is determined by multiplying the original user'shead posture angle q_(h) by a multiplicative inverse of the cameraposture angle q_(c) from the left hand. Moreover, each of the postureangles q_(h) and q_(c) is information on a posture angle of each of thehead and the camera measured using the above-mentioned third coordinatesystem as a reference.

[Math. 2]

q _(h) *=q _(c) ⁻¹ q _(h)   (2)

The image display system 100 in which an image capturing device providedwith an omnidirectional camera is mounted on the moving object device120 such as aircraft (or a helicopter and other flying objects), motorvehicle, and watercraft has been described above. Meanwhile, a videosee-through image display system 100 in which an omnidirectional camerais attached to the user who wears the head-mounted display 110illustrated in FIG. 15 and the user views a captured image through thehead-mounted display 110 is also conceivable. In this case, a displayangle of view depending on the user's head posture is clipped from anomnidirectional image captured by an omnidirectional camera and isdisplayed on the head-mounted display 110.

In such a video see-through image display system 100, the problem ofdiscrepancy between the user's head posture and its display angle ofview is caused by delay time from capturing by a camera to displaying aswell as by a variation in the camera's posture as illustrated in FIG.12. If the direction of the head does not match the display angle ofview, the user will view an unintended image, and thus is more likely tocause VR sickness.

Furthermore, in a see-through head-mounted display as illustrated inFIG. 15, if the mounting position and posture of a camera and theposition at which the user views a captured image (eye position) andposture of the user are in discrepancy, various problems occur as listedin the following items (1) to (3).

(1) Sense of distance is difficult to find as if the hand looks shorter

(2) Virtual reality sickness is easily get unless optical axis directionis adjusted

(3) Virtual reality sickness is easily get unless imaging viewing angleand display viewing angle match

The present inventors have found that the problem of delay ordiscrepancy between posture and viewing angle in the video see-throughimage display system 100 can be reduced by a combination of displaycorrection in consideration of camera posture and head motion trackingprediction.

For simplicity of description, a case where there is no delay time fromcapturing by a camera to displaying is considered. In this case, it isnecessary only to align a reference coordinate system of a camera with adisplay optical system (screen of a display unit) and to match it with aviewing angle to be presented to the user.

FIG. 16 illustrates a processing procedure for displaying an imageobtained by aligning an image captured by a camera with a referencecoordinate system of a display optical system and matching it with aviewing angle of the user (in this case, it is assumed that there is nodelay time from capturing to displaying).

The omnidirectional camera 1031 of the image capturing device 1030attached to the user captures an omnidirectional image (F1601).

The position of the omnidirectional camera 1301 relative to the user (orthe display device 1020 worn by the user) is assumed to be fixed. Inthis case, the user's head posture can be represented as a fixedparameter q_(t) using the camera's posture as a reference. Thus, adisplay angle of view depending on the fixation may be clipped from theimage captured by the camera (F1602 and F1603), and may be displayed(F1604). The camera and the head are moved together, and thus thecorrected user's head posture angle q_(h)′*=q_(c) ⁻¹q_(h) in the aboveformula (2) is typically kept at a constant value (independent of valuesof the posture angles q_(c) and q_(h)), which is called as the fixedparameter q_(t).

Furthermore, FIG. 17 illustrates a processing procedure for displayingan image obtained by aligning an image captured by a camera with areference coordinate system of a display optical system and matching itwith a viewing angle of the user in consideration of delay time fromcapturing to displaying.

The omnidirectional camera 1031 of the image capturing device 1030attached to the user captures an omnidirectional image (F1701).

Furthermore, the relative positional relationship of the omnidirectionalcamera 1301 to the user (or the display device 1020 worn by the user) isfixed, and the fixed parameter q_(t) for converting the camera's postureto the user's head posture is held (F1702). The fixed parameter q_(t) isdetermined by mechanical arrangement of the display optical system andthe camera capturing system.

Furthermore, in the head motion tracking device 1010, the sensor unit1011 detects a posture angle of the user's head, and the posture anglecalculation unit 1012 calculates a quaternion q_(h) indicating a postureangle of the user's head based on a result obtained by detection in thesensor unit 1011 (F1703), which is logged in association with timeinformation in acquiring the sensor-based information (F1704). Then, onthe basis of the time at which capturing is performed in F1701 and theestimation value q_(h) of the head posture logged in F1704, a headposture q_(hc) in capturing is estimated (i.e., interpolation orprediction for the capturing time is performed) (F1705). Furthermore,delay time δ from the current time to displaying an image on the displayoptical system is approximated (F1706). On the basis of the current time(F1707) and the estimation value q_(h) of the head posture logged inF1704, a head posture q_(h)′ at the time when the subsequent display isperformed (current time+delay time δ) is predicted (F1708). Moreover,the delay time δ is determined mainly depending on a drive frequency ofa display panel or the configuration of a panel drive circuit. Theprediction of the head posture by considering the delay time δ can beperformed, in one example, using the prediction algorithm disclosed inPatent Application No. 2013-268014, which is assigned to the presentapplicant.

In the image processing device 1040, on the basis of the fixed parameterq_(t) acquired in F1702, the head posture q_(hc) estimated in F1705, andthe prediction value q_(h)′ of the head posture predicted in F1708, acorrected parameter q_(t)* is calculated (F1709). More specifically, asshown in the following formula (3), the corrected parameter q_(t)* iscalculated by multiplying the fixed parameter q_(t) by correction termsq_(hc) ⁻¹ and q_(h)′, which are obtained by multiplying the predictionvalue q_(h)′ by a multiplicative inverse of the head posture q_(hc) incapturing from the left of the prediction value q_(h)′ of the headposture, from the right of the fixed parameter q_(t).

[Math. 3]

q* _(t) =q _(t) q _(hc) ⁻¹ q′ _(h)   (3)

Then, a display angle of view depending on the parameter q_(t)* isclipped from the captured image to render a free viewpoint image(F1710). The free viewpoint image rendered as described above istransmitted from the image processing device 1040 to the display device1020, and the display device 1020 displays it (F1711).

Furthermore, the present inventors have found that the problem of delayor discrepancy between posture and viewing angle in the videosee-through image display system 100 can be solved by a combination ofthree-dimensional reconfiguration technology including Visual SLAM as atypical example and the head motion tracking prediction. The Visual SLAMis technology capable of performing camera self-position estimation andcartography simultaneously under unknown environment. An example of theVisual SLAM can include integrated augmented reality technology,“SmartAR” (trademark of Sony Corporation).

In the imaging device 1040, the omnidirectional camera 1031 attached tothe user's head (or other body parts of the user) continues capturing,and thus can obtain time-series image data. In the image processingdevice 1040, a three-dimensional model of the surrounding environment isconstructed from the time-series image data using the Visual SLAMtechnology and the current position of the camera in thethree-dimensional model is found. Then, if the current eye position andposture of the user is predicted in consideration of the delay time δfrom capturing to displaying it on the display device 1020, the imageprocessing device 1040 renders an image captured by a virtual camera atthe predicted position, and the display device 1020 displays it.

FIG. 18 illustrates a processing procedure for displaying an image thatmatches the viewing angle of the user in consideration of the delay timefrom capturing to displaying using a combination of thethree-dimensional reconfiguration technology and the head motiontracking prediction.

The omnidirectional camera 1031 of the image capturing device 1030attached to the user captures an omnidirectional image (F1801).

In the image processing device 1040, a three-dimensional model M of thesurrounding environment is constructed from the time-series image datausing the Visual SLAM technology (F1802), a camera position p_(c) and acamera posture q_(c) in capturing in the three-dimensional model areestimated (F1803), and they are logged in association with each ofcapturing time information items (F1804).

Next, the delay time δ from the current time to displaying an image onthe display optical system is approximated (F1805). On the basis of thecurrent time (F1806) and the estimation values p_(c) and q_(c) of thecamera position and posture logged in F1804, a camera position p′_(c)and a camera posture q′_(c) at the time when the subsequent display isperformed are predicted (F1807). Furthermore, conversion parametersp_(t) and q_(t) for the position and posture of the omnidirectionalcamera 1031 and the position and posture of the user's eye are acquired(F1808). The conversion parameters p_(t) and q_(t) are fixed parametersdetermined by mechanical arrangement of the display optical system andthe camera capturing system. Moreover, the conversion parameter p_(t) isa three-dimensional vector that applies an offset of the coordinateposition, and the conversion parameter q_(t) is a quaternion thatrepresents a change in posture. Then, as shown in the following formula(4), a user's eye position p′_(h) and a user's eye posture q′_(h) at thetime when the subsequent display is performed are predicted from apredicted value p′_(c) of the camera position and a predicted valueq′_(c) of the camera posture at the time, using the conversionparameters p_(t) and q_(t) (F1809).

$\begin{matrix}\left\lbrack {{Math}.\mspace{14mu} 4} \right\rbrack & \; \\\left. \begin{matrix}{p_{h}^{\prime} = {p_{c}^{\prime} + p_{t}}} \\{q_{h}^{\prime} = {q_{c}^{\prime} \cdot q_{t}}}\end{matrix} \right\} & (4)\end{matrix}$

Then, in the image processing device 1040, image capturing data in thepredicted eye position p′_(h) and posture q′_(h) is rendered using thethree-dimensional model M of the surrounding environment constructed inF1802 (F1810). The free viewpoint image rendered as described above istransmitted from the image processing device 1040 to the display device1020, and the display device 1020 displays it (F1811).

As described above, according to the processing procedure shown in FIGS.17 and 18, it is possible to solve the problem of delay or discrepancybetween posture and viewing angle in the video see-through image displaysystem 100. Consequently, in a video see-through image, it is possibleto prevent the VR sickness by reducing misalignment such as optical axisdiscrepancy. Furthermore, the degree of freedom in camera selection orarrangement is improved, and thus effects as listed in the followingitems (1) to (4) can be obtained.

(1) Alignment between camera and eye's optical axis is unnecessary.

(2) Agreement between camera and eye's posture is unnecessary.

(3) Alignment between camera and eye's viewing angle is unnecessary.

(4) Arrangement of any number of cameras is possible.

Although the image display system 100 illustrated in FIG. 1 isconfigured to include the camera mounted on the moving object device 120and the head-mounted display 110 through which the user views an imagecaptured by the camera while being attached to the user's head, it canbe used as a camera system of the moving object device 120 that isremotely operated by the user using the controller 130.

In a typical remote operation system, any possible movement of themoving object device 120 can be operated using the controller 130.However, in the case where the remote operation is performed while animage captured by the camera mounted on the moving object device 120 isviewed, if the moving object device 120 performs a motion against theuser's intention or an unintended motion, the user is likely toexperience VR sickness.

Meanwhile, in the technology disclosed herein, the VR sickness isprevented by restricting the control degree of freedom of the movingobject device 120 by the user. In one example, the trajectory of themoving object device 120 in a space is specified, and only one ofposition, velocity, and acceleration on the trajectory is operatedremotely.

In the following, an embodiment in which the image display system 100 isoperated as a remote operation camera system capable of restricting thecontrol degree of freedom is described. In one example, the aerialmoving object device 120, such as helicopter and multicopter equippedwith three or more rotors, is assumed to be a target of remoteoperation, and the control degree of freedom is restricted only to astraight or circular motion. Then, the user remotely operates onlyvelocity or acceleration on a trajectory in a straight or circularmotion using an input device such as joystick that is made easy tooperate as the controller 130.

A remote operation using the controller 130 in a case where the controldegree of freedom of the moving object device 120 is restricted to astraight motion as illustrated in FIG. 19 is described. FIG. 20illustrates a processing procedure for controlling the moving objectdevice 120 in the case described above.

When a trajectory constraint mode with a restriction to a straightmotion is started (F2001), the trajectory of the moving object device120 is set (F2002). In the case where the trajectory of the movingobject device 120 is restricted to the straight motion, in one example,a straight-line trajectory as listed in the following item (1) or (2) isset.

(1) Straight line directed toward current direction of travel throughcurrent point

(2) Straight line connecting current point and destination point

If control is inputted from the controller 130 such as joystick (F2003),the control input is filtered, and other control inputs than aback-and-front direction are cut off (F2004).

Subsequently, the control input is converted to a position command,velocity command, or acceleration command along the straight-linetrajectory that is set in F2002 (F2005), and a command to maintain thetrajectory is conveyed to an automatic control system of the movingobject device 120 (F2006).

Next, a remote operation using the controller 130 in a case where thecontrol degree of freedom of the moving object device 120 is restrictedto a circular motion as illustrated in FIG. 21 is described. FIG. 22illustrates a processing procedure for controlling the moving objectdevice 120 in the case described above.

When a trajectory constraint mode with a restriction to a circularmotion is started (F2201), the trajectory of the moving object device120 is set (F2202). In the case where the trajectory of the movingobject device 120 is restricted to the circular motion, in one example,a circular trajectory as listed in the following item (1) or (2) is set.

(1) Circular motion through current point centered on destination point

(2) Circular motion in horizontal plane passing through current pointcentered on perpendicular axis intersecting destination point

If control is inputted from the controller 130 such as joystick (F2203),the control input is filtered, and other control inputs than aleft-and-right direction are cut off (F2204).

Subsequently, the control input is converted to a position command,velocity command, or acceleration command along the circular trajectorythat is set in F2202 (F2205). A command to maintain the trajectory isconveyed to an automatic control system of the moving object device 120and the posture of the moving object device 120 is controlled to bedirected toward a destination point (F2206).

INDUSTRIAL APPLICABILITY

The foregoing thus describes the technology disclosed in thisspecification in detail and with reference to specific embodiments.However, it is obvious that persons skilled in the art may makemodifications and substitutions to these embodiments without departingfrom the spirit of the technology disclosed in this specification.

Although the technology disclosed herein can be preferably applied inviewing an image captured by a remote camera mounted on a moving objectdevice or the like using an immersive head-mounted display, it iscertainly applicable to a see-through head-mounted display.

Furthermore, the technology disclosed herein is similarly applicable toa case where an image captured by a camera mounted on the main body of ahead mounted-display rather than a remote camera is viewed using videosee-through visualization.

Moreover, the technology disclosed herein is similarly applicable to acase where an image captured by a camera is viewed through a screen ofan information terminal such as smartphones and tablets fixed to thehead or face rather than a head-mounted display.

The technology described herein is preferably applicable to any type ofbinocular and monocular head-mounted displays.

In short, the technology disclosed in this specification has beendescribed by way of example, and it should not be construed as beinglimited to the description of this specification. The spirit of thetechnology disclosed in this specification should be determined inconsideration of the claims.

Additionally, the present technology may also be configured as below.

-   (1)

An information processing device including:

a head posture acquisition unit configured to acquire information onhead posture of a user;

a camera posture acquisition unit configured to acquire information onposture of a camera; and

an image rendering processor configured to generate an image to bedisplayed on a display device from an image captured by the camera basedon the head posture of the user and the posture of the camera, thedisplay device being fixed to a head or a face of the user.

-   (2)

The information processing device according to (1),

wherein the camera is mounted on a moving object device.

-   (3)

The information processing device according to any one of (1) and (2),

wherein the camera captures an omnidirectional image or a wide-angleimage, and

the image rendering processor corrects the head posture of the userusing the posture of the camera in capturing and generates an imageobtained by clipping an angle of view depending on the corrected headposture of the user from the image captured by the camera.

-   (4)

The information processing device according to (1),

wherein the camera is fixedly mounted on the head or the face of theuser.

-   (5)

The information processing device according to (4),

wherein the image rendering processor generates an image obtained byclipping an angle of view depending on a first conversion parameter fromthe image captured by the camera, the first conversion parameter beingused to convert the posture of the camera to the head posture of theuser.

-   (6)

The information processing device according to (5),

wherein the image rendering processor performs image generation usingthe head posture of the user at a time when the camera performscapturing and the head posture of the user predicted after a lapse ofdelay time until an image is displayed on the display device.

-   (7)

The information processing device according to (4),

wherein the image rendering processor constructs a three-dimensionalmodel of surrounding environment based on data of time-series imagecaptured by the camera, estimates a current position of the camera inthe three-dimensional model, predicts a position and posture of an eyeof the user after a lapse of delay time from capturing by the camera todisplaying on the display device using a second conversion parameterused to convert a position and posture of the camera to the position andposture of the eye of the user, and generates an image captured in apredicted position and posture of the eye from the three-dimensionalmodel.

-   (8)

The information processing device according to (1), further including:

a controller configured to remotely operate the moving object device;and

a filter configured to constrain a trajectory of the moving objectdevice and cut off an input other than the constrained trajectory fromthe controller,

wherein the input from the controller is converted to a positioncommand, a velocity command, or an acceleration command in a directionalong the constrained trajectory for transmitting to the moving objectdevice.

-   (9)

The information processing device according to (8),

wherein, when the trajectory of the moving object device is constrainedto a straight motion,

the filter cuts off an input other than a back-and-front direction fromthe controller, and

transmits a command to maintain a trajectory of the straight motion tothe moving object device.

-   (10)

The information processing device according to (9),

wherein the trajectory is set to a straight line toward a currentdirection of travel through a current point of the moving object or to astraight line connecting the current point of the moving object deviceand a destination point.

-   (11)

The information processing device according to (8),

wherein, when the trajectory of the moving object device is constrainedto a circular motion,

the filter cuts off an input other than a left-and-right direction fromthe controller, and

transmits a command to maintain a trajectory of the circular motion tothe moving object device and controls posture of the moving objectdevice to be directed toward a destination point.

-   (12)

The information processing device according to (11),

wherein the trajectory is set to a circular motion passing through acurrent point of the moving object device centered on a destinationpoint or to a circular motion in a horizontal plane through the currentpoint of the moving object device centered on a perpendicular axisintersecting a destination point.

-   (13)

A method of information processing, the method including:

a head posture acquisition step of acquiring information on head postureof a user;

a camera posture acquisition step of acquiring information on posture ofa camera; and

an image rendering processing step of generating an image to bedisplayed on a display device from an image captured by the camera basedon the head posture of the user and the posture of the camera, thedisplay device being fixed to a head or a face of the user.

-   (14)

An image display system including:

a camera;

a display device used while being fixed to a head or a face of a user;and

an image processing device configured to generate an image to bedisplayed on the display device from an image captured by the camerabased on the head posture of the user and the posture of the camera.

REFERENCE SIGNS LIST

-   100 image display system-   110 head-mounted display-   120 moving object device-   130 controller-   400 omnidirectional camera-   401-406 video camera-   410 image processor-   810 head motion tracking device-   811 sensor unit-   812 posture angle calculation unit-   813 communication unit-   820 display device-   821 first communication unit-   822 image rendering processor-   823 display unit-   824 second communication unit-   830 image capturing device-   831 omnidirectional camera-   832 sensor unit-   833 posture angle calculation unit-   834 communication unit-   1010 head motion tracking device-   1011 sensor unit-   1012 posture angle calculation unit-   1013 communication unit-   1020 display device-   1021 communication unit-   1023 display unit-   1030 image capturing device-   1031 omnidirectional camera-   1032 sensor unit-   1033 posture angle calculation unit-   1034 communication unit-   1040 image processing device-   1041 communication unit-   1042 image rendering processor

1. An information processing device comprising: a head postureacquisition unit configured to acquire information on head posture of auser; a camera posture acquisition unit configured to acquireinformation on posture of a camera; and an image rendering processorconfigured to generate an image to be displayed on a display device froman image captured by the camera based on the head posture of the userand the posture of the camera, the display device being fixed to a heador a face of the user.
 2. The information processing device according toclaim 1, wherein the camera is mounted on a moving object device.
 3. Theinformation processing device according to claim 1, wherein the cameracaptures an omnidirectional image or a wide-angle image, and the imagerendering processor corrects the head posture of the user using theposture of the camera in capturing and generates an image obtained byclipping an angle of view depending on the corrected head posture of theuser from the image captured by the camera.
 4. The informationprocessing device according to claim 1, wherein the camera is fixedlymounted on the head or the face of the user.
 5. The informationprocessing device according to claim 4, wherein the image renderingprocessor generates an image obtained by clipping an angle of viewdepending on a first conversion parameter from the image captured by thecamera, the first conversion parameter being used to convert the postureof the camera to the head posture of the user.
 6. The informationprocessing device according to claim 5, wherein the image renderingprocessor performs image generation using the head posture of the userat a time when the camera performs capturing and the head posture of theuser predicted after a lapse of delay time until an image is displayedon the display device.
 7. The information processing device according toclaim 4, wherein the image rendering processor constructs athree-dimensional model of surrounding environment based on data oftime-series image captured by the camera, estimates a current positionof the camera in the three-dimensional model, predicts a position andposture of an eye of the user after a lapse of delay time from capturingby the camera to displaying on the display device using a secondconversion parameter used to convert a position and posture of thecamera to the position and posture of the eye of the user, and generatesan image captured in a predicted position and posture of the eye fromthe three-dimensional model.
 8. The information processing deviceaccording to claim 1, further comprising: a controller configured toremotely operate the moving object device; and a filter configured toconstrain a trajectory of the moving object device and cut off an inputother than the constrained trajectory from the controller, wherein theinput from the controller is converted to a position command, a velocitycommand, or an acceleration command in a direction along the constrainedtrajectory for transmitting to the moving object device.
 9. Theinformation processing device according to claim 8, wherein, when thetrajectory of the moving object device is constrained to a straightmotion, the filter cuts off an input other than a back-and-frontdirection from the controller, and transmits a command to maintain atrajectory of the straight motion to the moving object device.
 10. Theinformation processing device according to claim 9, wherein thetrajectory is set to a straight line toward a current direction oftravel through a current point of the moving object or to a straightline connecting the current point of the moving object device and adestination point.
 11. The information processing device according toclaim 8, wherein, when the trajectory of the moving object device isconstrained to a circular motion, the filter cuts off an input otherthan a left-and-right direction from the controller, and transmits acommand to maintain a trajectory of the circular motion to the movingobject device and controls posture of the moving object device to bedirected toward a destination point.
 12. The information processingdevice according to claim 11, wherein the trajectory is set to acircular motion passing through a current point of the moving objectdevice centered on a destination point or to a circular motion in ahorizontal plane through the current point of the moving object devicecentered on a perpendicular axis intersecting a destination point.
 13. Amethod of information processing, the method comprising: a head postureacquisition step of acquiring information on head posture of a user; acamera posture acquisition step of acquiring information on posture of acamera; and an image rendering processing step of generating an image tobe displayed on a display device from an image captured by the camerabased on the head posture of the user and the posture of the camera, thedisplay device being fixed to a head or a face of the user.
 14. An imagedisplay system comprising: a camera; a display device used while beingfixed to a head or a face of a user; a head motion tracking deviceconfigured to measure head posture of the user; and an image processingdevice configured to generate an image to be displayed on the displaydevice from an image captured by the camera based on the head posture ofthe user and the posture of the camera.